Digital cameras that are able to capture images both in the visible spectrum and in an invisible light spectrum are useful in many contexts such as computer vision, image processing, and surveillance systems. The visible light (or “color”) spectrum may include, for example, red, green, and blue (RGB) light. The invisible light spectrum may include, for example, a portion of the infrared spectrum. The color information can be used to provide photo-realistic images for user viewing or to perform computer vision tasks in situations illuminated in the visible spectra, such as in sunlight or under artificial illumination. The infrared information may be exploited in situations in which an application may benefit from full illumination control without affecting ambient visible illumination, such as video conferencing in low light scenarios and depth reconstruction systems supported by a coded light source (e.g., a projection source configured to project a pattern such as dots or lines onto a scene).
Sensors that are capable of detecting visible (e.g., RGB) and invisible (e.g., IR) information may be standard charged coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensors. FIGS. 1A and 1B compare the architecture of an RGB-IR image sensor with that of a conventional RGB image sensor. Generally, as shown in FIG. 1B, a conventional RGB camera sensor includes pixels arranged in a “Bayer layout” or “RGBG layout,” which is 50% green, 25% red, and 25% blue. Band pass filters (or “micro filters”) are placed above individual photodiodes for each of the green, red, and blue wavelengths in accordance with the Bayer layout. Generally, a conventional RGB camera sensor also includes an infrared (IR) filter or IR cut-off filter (formed, e.g., as part of the lens or as a coating on the entire chip) which further blocks signals in an IR portion of electromagnetic spectrum, as illustrated by the dashed line in FIG. 1B.
An RGB-IR sensor as illustrated in FIG. 1A is substantially similar to a conventional RGB sensor, but may include different color filters. For example, as shown in FIG. 1A, in an RGB-IR sensor, one of the green filters in every group of four photodiodes is replaced with an IR band-pass filter (or micro filter) to create a layout that is 25% green, 25% red, 25% blue, and 25% infrared, where the infrared pixels are intermingled among the visible light pixels. In addition, the IR cut-off filter may be omitted from the RGB-IR sensor, the IR cut-off filter may be located only over the pixels that detect red, green, and blue light, or the IR filter can be designed to pass visible light as well as light in a particular wavelength interval (e.g., 840-860 nm). An image sensor capable of capturing light in multiple portions or bands or spectral bands of the electromagnetic spectrum (e.g., red, blue, green, and infrared light) will be referred to herein as a “multi-channel” image sensor.
Some multi-channel image sensors allow substantially simultaneous or concurrent collection of both visible and invisible light in a scene. As such, when the scene is illuminated by a light source capable of emitting invisible light, the multi-channel sensor can provide both visible light information and invisible light from the reflected patterns for computer vision applications.
However, in many circumstances the luminance of the light in the visible light band (e.g., detected in a visible light channel) may be significantly different from the luminance of the light in the invisible light band (e.g., detected in an invisible light channel) and therefore, any given set of capture parameters (or exposure parameters) such as exposure time (shutter speed), aperture (f-stop), gain (ISO), and white balance, may fail to yield good results for either the visible channel (or channels) or the invisible channel (or channels). This may be especially true in circumstances where the RGB-IR sensor is coupled to an infrared illumination source that provides controlled illumination of the scene.
Generally, there are two different levels of optical filtering on a sensor: a coating filter, which is distributed on the system optics and which affects the measurements taken by all the pixels, and a per pixel filter which controls the particular portion of the spectrum (e.g., the particular band) transmitted to each pixel (e.g., filters to transmit the blue band to the blue pixels, the red band to red pixels, etc.).
FIG. 1C is a graph illustrating the transmissivity of optical filters that may be implemented on different pixels of a multi-channel camera sensor. As shown in FIG. 1C, there is cross-talk between the red, green, and blue channels and the infrared channel. In particular, the infrared channel has a non-zero response in the visible light range (e.g., about 400 nm to about 700 nm) and therefore the infrared channel would be affected by visible light illumination. Similarly, all of the red, green, and blue filters pass significant energy in the infrared band (e.g., about 760 nm to 1000 nm), so the red, blue, and green channels are affected by infrared illumination, that is, the red, blue, and green pixels would also detect infrared light in a scene. The effect of infrared illumination on the visible light channels would be particularly noticeable in situations where an infrared light source projected a pattern of dots or lines onto the scene because the pattern would appear in the captured light image despite being invisible to the naked eye.